Skip to content

branch-3.0: [fix](cloud) fix routine load loss data when fe master node restart #46149#46566

Merged
liaoxin01 merged 1 commit intobranch-3.0from
auto-pick-46149-branch-3.0
Jan 8, 2025
Merged

branch-3.0: [fix](cloud) fix routine load loss data when fe master node restart #46149#46566
liaoxin01 merged 1 commit intobranch-3.0from
auto-pick-46149-branch-3.0

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 7, 2025

Cherry-picked from #46149

@sollhui
[fix](cloud) fix routine load loss data when fe master node restart (#…
4258d74 
…46149)


In cloud mode, routine load loss data when fe master node restart.

When updating progress, in order to avoid small values covering large
values, we introduced pr #39313, Due
to the pr that the routine load replays progress metadata by first
obtaining the set default offset and then pulling metadata from meta
service to update the local value, if the metadata pulled from meta
service is not larger than the set default offset, the correct value
cannot be assigned to memory.

To solve this problem, pulling metadata from meta service when restart,
determine whether to obtain default offset from Kafka based on the
pulled value.
@Thearas
Copy link
Contributor

Thearas commented Jan 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 7, 2025
@dataroaring dataroaring reopened this Jan 7, 2025
@Thearas
Copy link
Contributor

Thearas commented Jan 7, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 7, 2025

TPC-H: Total hot run time: 40611 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4258d743423be19007c3aa1f83b45162d44311a0, data reload: false

------ Round 1 ----------------------------------
q1	17597	7371	7252	7252
q2	2051	184	164	164
q3	10984	1038	1147	1038
q4	10586	768	820	768
q5	7835	2869	2882	2869
q6	239	147	143	143
q7	960	626	604	604
q8	9353	1924	2028	1924
q9	6556	6380	6303	6303
q10	7065	2299	2278	2278
q11	467	268	266	266
q12	406	217	215	215
q13	17784	2958	3017	2958
q14	242	226	208	208
q15	595	523	531	523
q16	695	609	620	609
q17	958	615	545	545
q18	7188	6599	6765	6599
q19	1407	1069	1018	1018
q20	462	208	197	197
q21	3975	3338	3127	3127
q22	1087	1008	1003	1003
Total cold run time: 108492 ms
Total hot run time: 40611 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7230	7245	7238	7238
q2	326	224	249	224
q3	2908	2934	2972	2934
q4	1981	1788	1830	1788
q5	5718	5679	5710	5679
q6	230	140	138	138
q7	2329	1798	1840	1798
q8	3387	3526	3500	3500
q9	8807	8834	8812	8812
q10	3555	3554	3528	3528
q11	589	502	490	490
q12	839	565	612	565
q13	9451	3195	3180	3180
q14	306	267	275	267
q15	589	520	542	520
q16	703	689	675	675
q17	1844	1602	1594	1594
q18	8253	7855	7691	7691
q19	1642	1538	1581	1538
q20	2140	1897	1834	1834
q21	5571	5300	5264	5264
q22	1168	1059	1029	1029
Total cold run time: 69566 ms
Total hot run time: 60286 ms

@doris-robot
Copy link

doris-robot commented Jan 7, 2025

TPC-DS: Total hot run time: 197602 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4258d743423be19007c3aa1f83b45162d44311a0, data reload: false

query1	1281	927	908	908
query2	6229	2115	2062	2062
query3	10838	4278	4294	4278
query4	66098	28859	23406	23406
query5	4970	452	461	452
query6	413	182	176	176
query7	5515	311	305	305
query8	311	228	231	228
query9	8701	2659	2658	2658
query10	446	276	260	260
query11	16859	15194	15717	15194
query12	161	101	105	101
query13	1447	447	435	435
query14	10385	7252	7376	7252
query15	196	192	176	176
query16	7068	533	504	504
query17	1058	584	587	584
query18	1289	323	315	315
query19	211	167	159	159
query20	117	110	112	110
query21	205	108	109	108
query22	4886	4442	4371	4371
query23	34835	34269	34094	34094
query24	6041	2878	2933	2878
query25	538	425	432	425
query26	710	170	166	166
query27	2009	359	371	359
query28	4483	2519	2463	2463
query29	702	481	458	458
query30	249	166	174	166
query31	967	830	852	830
query32	70	56	58	56
query33	460	283	279	279
query34	916	511	534	511
query35	823	735	747	735
query36	1085	955	930	930
query37	125	76	74	74
query38	4087	4078	4081	4078
query39	1487	1531	1490	1490
query40	195	101	98	98
query41	51	47	50	47
query42	110	99	98	98
query43	534	497	483	483
query44	1165	845	854	845
query45	181	168	170	168
query46	1162	746	745	745
query47	2041	1946	1950	1946
query48	484	411	390	390
query49	724	382	383	382
query50	847	436	434	434
query51	7404	7111	7214	7111
query52	105	90	89	89
query53	248	183	176	176
query54	546	450	450	450
query55	77	75	80	75
query56	260	252	242	242
query57	1222	1103	1085	1085
query58	210	200	204	200
query59	3355	3193	2960	2960
query60	285	277	254	254
query61	116	110	113	110
query62	770	667	677	667
query63	207	192	190	190
query64	1371	654	629	629
query65	3261	3173	3183	3173
query66	630	303	293	293
query67	15942	15788	15789	15788
query68	3670	595	587	587
query69	431	279	267	267
query70	1220	1059	1130	1059
query71	350	247	265	247
query72	6373	4104	4013	4013
query73	748	362	357	357
query74	10334	8895	9024	8895
query75	3360	2636	2670	2636
query76	1854	1043	1116	1043
query77	499	277	276	276
query78	10460	9649	9558	9558
query79	1684	589	607	589
query80	1442	415	408	408
query81	528	246	240	240
query82	1307	118	119	118
query83	270	146	145	145
query84	287	81	81	81
query85	1019	302	285	285
query86	433	302	307	302
query87	4479	4305	4262	4262
query88	3954	2403	2373	2373
query89	421	286	284	284
query90	1933	185	186	185
query91	175	150	149	149
query92	61	52	50	50
query93	2124	547	547	547
query94	842	306	299	299
query95	348	264	253	253
query96	608	277	282	277
query97	3343	3240	3176	3176
query98	222	210	207	207
query99	1651	1300	1272	1272
Total cold run time: 318204 ms
Total hot run time: 197602 ms

@doris-robot
Copy link

doris-robot commented Jan 7, 2025

ClickBench: Total hot run time: 33.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4258d743423be19007c3aa1f83b45162d44311a0, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.53	0.52	0.53
query6	1.14	0.72	0.72
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.55	0.49	0.51
query10	0.56	0.54	0.55
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.62
query14	2.96	2.93	3.07
query15	0.90	0.82	0.83
query16	0.38	0.39	0.39
query17	1.07	1.05	1.03
query18	0.23	0.22	0.22
query19	1.88	1.84	1.91
query20	0.01	0.01	0.02
query21	15.35	0.60	0.57
query22	2.64	2.80	2.53
query23	17.19	0.93	0.93
query24	3.28	1.12	0.87
query25	0.21	0.09	0.07
query26	0.40	0.13	0.14
query27	0.04	0.04	0.03
query28	10.58	1.10	1.06
query29	12.55	3.28	3.31
query30	0.24	0.06	0.06
query31	2.87	0.40	0.39
query32	3.23	0.46	0.48
query33	3.02	3.00	3.07
query34	16.66	4.49	4.48
query35	4.52	4.56	4.62
query36	0.65	0.50	0.49
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.09 s
Total hot run time: 33.81 s

@liaoxin01 liaoxin01 merged commit 197f860 into branch-3.0 Jan 8, 2025
21 of 22 checks passed
@github-actions github-actions bot deleted the auto-pick-46149-branch-3.0 branch January 8, 2025 03:55
@gavinchou gavinchou mentioned this pull request Feb 18, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @liaoxin01 @dataroaring @sollhui